AutoML Leaderboard

Best model	name	model_type	metric_type	metric_value	train_time
	1_Baseline	Baseline	logloss	1.35895	1.42
	2_DecisionTree	Decision Tree	logloss	0.638737	26.51
	3_Linear	Linear	logloss	0.737755	15.63
the best	4_Default_Xgboost	Xgboost	logloss	0.175651	40.78
	5_Default_NeuralNetwork	Neural Network	logloss	0.364518	9.07
	6_Default_RandomForest	Random Forest	logloss	0.527478	18.71
	Ensemble	Ensemble	logloss	0.175651	0.52

AutoML Performance

AutoML Performance Boxplot

Features Importance

Spearman Correlation of Models

Summary of 6_Default_RandomForest

<< Go back

Random Forest

• n_jobs: -1
• criterion: gini
• max_features: 0.9
• min_samples_split: 30
• max_depth: 4
• eval_metric_name: logloss
• num_class: 6
• explain_level: 2

Validation

• validation_type: split
• train_ratio: 0.75
• shuffle: True
• stratify: True

Optimized metric

logloss

Training time

17.5 seconds

Metric details

	2 yr Undergrad	4 yr Undergrad	5 yr Undergrad	Grad	STAP	Unknown	accuracy	macro avg	weighted avg	logloss
precision	0.754829	0.870036	0.652397	0.996429	0	0	0.789096	0.545615	0.801755	0.527478
recall	0.824675	0.651351	0.858108	1	0	0	0.789096	0.555689	0.789096	0.527478
f1-score	0.788208	0.744977	0.741245	0.998211	0	0	0.789096	0.54544	0.786434	0.527478
support	616	740	444	279	7	5	0.789096	2091	2091	0.527478

Confusion matrix

	Predicted as 2 yr Undergrad	Predicted as 4 yr Undergrad	Predicted as 5 yr Undergrad	Predicted as Grad	Predicted as STAP	Predicted as Unknown
Labeled as 2 yr Undergrad	508	19	89	0	0	0
Labeled as 4 yr Undergrad	150	482	107	1	0	0
Labeled as 5 yr Undergrad	15	48	381	0	0	0
Labeled as Grad	0	0	0	279	0	0
Labeled as STAP	0	0	7	0	0	0
Labeled as Unknown	0	5	0	0	0	0

Learning curves

Permutation-based Importance

Confusion Matrix

Normalized Confusion Matrix

ROC Curve

Precision Recall Curve

SHAP Importance

SHAP Dependence plots

Dependence 2 yr Undergrad (Fold 1)

Dependence 4 yr Undergrad (Fold 1)

Dependence 5 yr Undergrad (Fold 1)

Dependence Grad (Fold 1)

Dependence STAP (Fold 1)

Dependence Unknown (Fold 1)

SHAP Decision plots

Worst decisions for selected sample 1 (Fold 1)

Worst decisions for selected sample 2 (Fold 1)

Worst decisions for selected sample 3 (Fold 1)

Worst decisions for selected sample 4 (Fold 1)

Best decisions for selected sample 1 (Fold 1)

Best decisions for selected sample 2 (Fold 1)

Best decisions for selected sample 3 (Fold 1)

Best decisions for selected sample 4 (Fold 1)

<< Go back

Summary of 2_DecisionTree

<< Go back

Decision Tree

• n_jobs: -1
• criterion: gini
• max_depth: 3
• num_class: 6
• explain_level: 2

Validation

• validation_type: split
• train_ratio: 0.75
• shuffle: True
• stratify: True

Optimized metric

logloss

Training time

25.4 seconds

Metric details

	2 yr Undergrad	4 yr Undergrad	5 yr Undergrad	Grad	STAP	Unknown	accuracy	macro avg	weighted avg	logloss
precision	0.768362	0.760243	0.57971	0.996429	0	0	0.740316	0.517457	0.751452	0.638737
recall	0.662338	0.677027	0.810811	1	0	0	0.740316	0.525029	0.740316	0.638737
f1-score	0.711421	0.716226	0.676056	0.998211	0	0	0.740316	0.516986	0.739796	0.638737
support	616	740	444	279	7	5	0.740316	2091	2091	0.638737

Confusion matrix

	Predicted as 2 yr Undergrad	Predicted as 4 yr Undergrad	Predicted as 5 yr Undergrad	Predicted as Grad	Predicted as STAP	Predicted as Unknown
Labeled as 2 yr Undergrad	408	75	133	0	0	0
Labeled as 4 yr Undergrad	117	501	121	1	0	0
Labeled as 5 yr Undergrad	6	78	360	0	0	0
Labeled as Grad	0	0	0	279	0	0
Labeled as STAP	0	0	7	0	0	0
Labeled as Unknown	0	5	0	0	0	0

Learning curves

Decision Tree

Tree #1

Rules

if (TAP Recipient Dollars <= 444314.375) and (TAP College Code <= 4987.5) and (TAP Sector Group <= 4.5) then class: 5 yr Undergrad (proba: 59.08%) | based on 1,877 samples

if (TAP Recipient Dollars > 444314.375) and (TAP College Code > 1249.5) and (TAP Sector Group > 0.5) then class: 2 yr Undergrad (proba: 72.19%) | based on 1,255 samples

if (TAP Recipient Dollars > 444314.375) and (TAP College Code <= 1249.5) and (Federal School Code <= 8294.0) then class: 4 yr Undergrad (proba: 98.55%) | based on 1,037 samples

if (TAP Recipient Dollars <= 444314.375) and (TAP College Code > 4987.5) and (TAP College Code <= 5985.0) then class: Grad (proba: 100.0%) | based on 836 samples

if (TAP Recipient Dollars <= 444314.375) and (TAP College Code > 4987.5) and (TAP College Code > 5985.0) then class: 4 yr Undergrad (proba: 47.03%) | based on 708 samples

if (TAP Recipient Dollars <= 444314.375) and (TAP College Code <= 4987.5) and (TAP Sector Group > 4.5) then class: 2 yr Undergrad (proba: 90.17%) | based on 295 samples

if (TAP Recipient Dollars > 444314.375) and (TAP College Code > 1249.5) and (TAP Sector Group <= 0.5) then class: 4 yr Undergrad (proba: 71.27%) | based on 181 samples

if (TAP Recipient Dollars > 444314.375) and (TAP College Code <= 1249.5) and (Federal School Code > 8294.0) then class: 4 yr Undergrad (proba: 64.63%) | based on 82 samples

Permutation-based Importance

Confusion Matrix

Normalized Confusion Matrix

ROC Curve

Precision Recall Curve

SHAP Importance

SHAP Dependence plots

Dependence 2 yr Undergrad (Fold 1)

Dependence 4 yr Undergrad (Fold 1)

Dependence 5 yr Undergrad (Fold 1)

Dependence Grad (Fold 1)

Dependence STAP (Fold 1)

Dependence Unknown (Fold 1)

SHAP Decision plots

Worst decisions for selected sample 1 (Fold 1)

Worst decisions for selected sample 2 (Fold 1)

Worst decisions for selected sample 3 (Fold 1)

Worst decisions for selected sample 4 (Fold 1)

Best decisions for selected sample 1 (Fold 1)

Best decisions for selected sample 2 (Fold 1)

Best decisions for selected sample 3 (Fold 1)

Best decisions for selected sample 4 (Fold 1)

<< Go back

Summary of 5_Default_NeuralNetwork

<< Go back

Neural Network

• n_jobs: -1
• dense_1_size: 32
• dense_2_size: 16
• learning_rate: 0.05
• num_class: 6
• explain_level: 2

Validation

• validation_type: split
• train_ratio: 0.75
• shuffle: True
• stratify: True

Optimized metric

logloss

Training time

8.1 seconds

Metric details

	2 yr Undergrad	4 yr Undergrad	5 yr Undergrad	Grad	STAP	Unknown	accuracy	macro avg	weighted avg	logloss
precision	0.869085	0.889201	0.744589	0.975177	0	0	0.862745	0.579675	0.858937	0.364518
recall	0.894481	0.856757	0.774775	0.985663	0	0	0.862745	0.585279	0.862745	0.364518
f1-score	0.8816	0.872677	0.759382	0.980392	0	0	0.862745	0.582342	0.860613	0.364518
support	616	740	444	279	7	5	0.862745	2091	2091	0.364518

Confusion matrix

	Predicted as 2 yr Undergrad	Predicted as 4 yr Undergrad	Predicted as 5 yr Undergrad	Predicted as Grad	Predicted as STAP	Predicted as Unknown
Labeled as 2 yr Undergrad	551	24	41	0	0	0
Labeled as 4 yr Undergrad	31	634	70	5	0	0
Labeled as 5 yr Undergrad	50	48	344	2	0	0
Labeled as Grad	0	0	4	275	0	0
Labeled as STAP	2	2	3	0	0	0
Labeled as Unknown	0	5	0	0	0	0

Learning curves

Permutation-based Importance

Confusion Matrix

Normalized Confusion Matrix

ROC Curve

Precision Recall Curve

<< Go back

Summary of 4_Default_Xgboost

<< Go back

Extreme Gradient Boosting (Xgboost)

• n_jobs: -1
• objective: multi:softprob
• eta: 0.075
• max_depth: 6
• min_child_weight: 1
• subsample: 1.0
• colsample_bytree: 1.0
• eval_metric: mlogloss
• num_class: 6
• explain_level: 2

Validation

• validation_type: split
• train_ratio: 0.75
• shuffle: True
• stratify: True

Optimized metric

logloss

Training time

39.6 seconds

Metric details

	2 yr Undergrad	4 yr Undergrad	5 yr Undergrad	Grad	STAP	Unknown	accuracy	macro avg	weighted avg	logloss
precision	0.946429	0.959722	0.880342	0.996429	1	0.833333	0.942611	0.936042	0.943681	0.175651
recall	0.946429	0.933784	0.927928	1	0.142857	1	0.942611	0.825166	0.942611	0.175651
f1-score	0.946429	0.946575	0.903509	0.998211	0.25	0.909091	0.942611	0.825636	0.941856	0.175651
support	616	740	444	279	7	5	0.942611	2091	2091	0.175651

Confusion matrix

	Predicted as 2 yr Undergrad	Predicted as 4 yr Undergrad	Predicted as 5 yr Undergrad	Predicted as Grad	Predicted as STAP	Predicted as Unknown
Labeled as 2 yr Undergrad	583	8	25	0	0	0
Labeled as 4 yr Undergrad	16	691	31	1	0	1
Labeled as 5 yr Undergrad	15	17	412	0	0	0
Labeled as Grad	0	0	0	279	0	0
Labeled as STAP	2	4	0	0	1	0
Labeled as Unknown	0	0	0	0	0	5

Learning curves

Permutation-based Importance

Confusion Matrix

Normalized Confusion Matrix

ROC Curve

Precision Recall Curve

SHAP Importance

SHAP Dependence plots

Dependence 2 yr Undergrad (Fold 1)

Dependence 4 yr Undergrad (Fold 1)

Dependence 5 yr Undergrad (Fold 1)

Dependence Grad (Fold 1)

Dependence STAP (Fold 1)

Dependence Unknown (Fold 1)

SHAP Decision plots

Worst decisions for selected sample 1 (Fold 1)

Worst decisions for selected sample 2 (Fold 1)

Worst decisions for selected sample 3 (Fold 1)

Worst decisions for selected sample 4 (Fold 1)

Best decisions for selected sample 1 (Fold 1)

Best decisions for selected sample 2 (Fold 1)

Best decisions for selected sample 3 (Fold 1)

Best decisions for selected sample 4 (Fold 1)

<< Go back

Summary of 1_Baseline

<< Go back

Baseline Classifier (Baseline)

• n_jobs: -1
• num_class: 6
• explain_level: 2

Validation

• validation_type: split
• train_ratio: 0.75
• shuffle: True
• stratify: True

Optimized metric

logloss

Training time

0.6 seconds

Metric details

	2 yr Undergrad	4 yr Undergrad	5 yr Undergrad	Grad	STAP	Unknown	accuracy	macro avg	weighted avg	logloss
precision	0	0.353898	0	0	0	0	0.353898	0.0589829	0.125244	1.35895
recall	0	1	0	0	0	0	0.353898	0.166667	0.353898	1.35895
f1-score	0	0.522783	0	0	0	0	0.353898	0.0871306	0.185012	1.35895
support	616	740	444	279	7	5	0.353898	2091	2091	1.35895

Confusion matrix

	Predicted as 2 yr Undergrad	Predicted as 4 yr Undergrad	Predicted as 5 yr Undergrad	Predicted as Grad	Predicted as STAP	Predicted as Unknown
Labeled as 2 yr Undergrad	0	616	0	0	0	0
Labeled as 4 yr Undergrad	0	740	0	0	0	0
Labeled as 5 yr Undergrad	0	444	0	0	0	0
Labeled as Grad	0	279	0	0	0	0
Labeled as STAP	0	7	0	0	0	0
Labeled as Unknown	0	5	0	0	0	0

Learning curves

Confusion Matrix

Normalized Confusion Matrix

ROC Curve

Precision Recall Curve

<< Go back

Summary of Ensemble

<< Go back

Ensemble structure

Model	Weight
4_Default_Xgboost	1

Metric details

	2 yr Undergrad	4 yr Undergrad	5 yr Undergrad	Grad	STAP	Unknown	accuracy	macro avg	weighted avg	logloss
precision	0.946429	0.959722	0.880342	0.996429	1	0.833333	0.942611	0.936042	0.943681	0.175651
recall	0.946429	0.933784	0.927928	1	0.142857	1	0.942611	0.825166	0.942611	0.175651
f1-score	0.946429	0.946575	0.903509	0.998211	0.25	0.909091	0.942611	0.825636	0.941856	0.175651
support	616	740	444	279	7	5	0.942611	2091	2091	0.175651

Confusion matrix

	Predicted as 2 yr Undergrad	Predicted as 4 yr Undergrad	Predicted as 5 yr Undergrad	Predicted as Grad	Predicted as STAP	Predicted as Unknown
Labeled as 2 yr Undergrad	583	8	25	0	0	0
Labeled as 4 yr Undergrad	16	691	31	1	0	1
Labeled as 5 yr Undergrad	15	17	412	0	0	0
Labeled as Grad	0	0	0	279	0	0
Labeled as STAP	2	4	0	0	1	0
Labeled as Unknown	0	0	0	0	0	5

Learning curves

Confusion Matrix

Normalized Confusion Matrix

ROC Curve

Precision Recall Curve

<< Go back

Summary of 3_Linear

<< Go back

Logistic Regression (Linear)

• n_jobs: -1
• num_class: 6
• explain_level: 2

Validation

• validation_type: split
• train_ratio: 0.75
• shuffle: True
• stratify: True

Optimized metric

logloss

Training time

14.5 seconds

Metric details

	2 yr Undergrad	4 yr Undergrad	5 yr Undergrad	Grad	STAP	Unknown	accuracy	macro avg	weighted avg	logloss
precision	0.649819	0.762821	0.639432	0.773639	0	0	0.7011	0.470952	0.700396	0.737755
recall	0.584416	0.643243	0.810811	0.967742	0	0	0.7011	0.501035	0.7011	0.737755
f1-score	0.615385	0.697947	0.714995	0.859873	0	0	0.7011	0.481367	0.694845	0.737755
support	616	740	444	279	7	5	0.7011	2091	2091	0.737755

Confusion matrix

	Predicted as 2 yr Undergrad	Predicted as 4 yr Undergrad	Predicted as 5 yr Undergrad	Predicted as Grad	Predicted as STAP	Predicted as Unknown
Labeled as 2 yr Undergrad	360	139	98	19	0	0
Labeled as 4 yr Undergrad	138	476	91	34	0	1
Labeled as 5 yr Undergrad	50	8	360	26	0	0
Labeled as Grad	1	1	7	270	0	0
Labeled as STAP	0	0	7	0	0	0
Labeled as Unknown	5	0	0	0	0	0

Learning curves

Coefficients

Coefficients learner #1

	2 yr Undergrad	4 yr Undergrad	5 yr Undergrad	Grad	STAP	Unknown
intercept	6.06943	6.27401	0.326364	-2.90855	-2.346	-7.41525
Academic Year	-0.273755	0.15808	0.189285	-1.38461	-0.857558	2.16856
TAP College Code	-0.633708	-0.628574	-0.927317	2.42116	-0.980313	0.748751
Federal School Code	0.630413	0.379101	0.0386286	-0.00863484	-0.344665	-0.694843
TAP College Name	-0.157055	-0.0477283	-0.0106782	0.433878	-0.209735	-0.00868053
Sector Type	0.926113	0.407807	1.13307	-2.63263	0.177392	-0.0117496
TAP Sector Group	1.07061	0.623883	0.437183	-4.19718	0.215916	1.84959
TAP Recipient Headcount	7.24263	-3.42542	-3.7786	0.72071	-0.598216	-0.161104
TAP Recipient FTEs	-3.56634	7.03864	-4.50686	1.89698	-0.675325	-0.187093
TAP Recipient Dollars	3.33042	3.84856	2.39031	-8.64638	-0.623034	-0.299876

Permutation-based Importance

Confusion Matrix

Normalized Confusion Matrix

ROC Curve

Precision Recall Curve

SHAP Importance

SHAP Dependence plots

Dependence 2 yr Undergrad (Fold 1)

Dependence 4 yr Undergrad (Fold 1)

Dependence 5 yr Undergrad (Fold 1)

Dependence Grad (Fold 1)

Dependence STAP (Fold 1)

Dependence Unknown (Fold 1)

SHAP Decision plots

Worst decisions for selected sample 1 (Fold 1)

Worst decisions for selected sample 2 (Fold 1)

Worst decisions for selected sample 3 (Fold 1)

Worst decisions for selected sample 4 (Fold 1)

Best decisions for selected sample 1 (Fold 1)

Best decisions for selected sample 2 (Fold 1)

Best decisions for selected sample 3 (Fold 1)

Best decisions for selected sample 4 (Fold 1)

<< Go back